Multiple gaze dependent illumination sources for retinal eye tracking

ABSTRACT

Various implementations disclosed herein include devices, systems, and methods that provide retinal imaging-based gaze tracking In some implementations, a user&#39;s gaze is tracked based on a retinal imaging technique that selectively uses a subset of multiple light sources ( 222, 310, 610 ) that illuminate different portions of the user&#39;s retina ( 352 ). In some implementations, a method ( 700 ) includes selecting ( 710 ) a subset of light sources, where the subset of the light sources includes less than all of the light sources. In some implementations, one or more portions of a retina ( 352 ) are illuminated ( 720 ) by producing light using the subset of the light sources. In some implementations, sensor data is received ( 730 ) at a sensor ( 224, 340, 814 ), the sensor data corresponding to the light detected during retinal imaging, and an eye characteristic is determined ( 740 ) based on the sensor data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is the national stage of InternationalApplication No. PCT/US2021/049783 filed on Sep. 10, 2021, which claimsthe benefit of U.S. Provisional Patent Application No. 63/081,545 filedon Sep. 22, 2020, entitled “MULTIPLE GAZE DEPENDENT ILLUMINATION SOURCESFOR RETINAL EYE TRACKING,” each of which is incorporated herein by thisreference in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to providing retinalimaging-based eye tracking of a user of an electronic device and, inparticular, to systems, methods, and devices that provide retinalimaging-based eye tracking of a user of an electronic device usingmultiple illumination sources.

BACKGROUND

Retinal imaging-based tracking systems generally have both anillumination source to direct light towards the retina and an imagesensor to generate images including light reflected from the retina.Drawbacks of such tracking systems include the relatively significantamount of power used by the illumination sources and unwantedreflections (glare) from eye surfaces other than the retina, e.g.,cornea, crystalline lens, etc.

SUMMARY

Various implementations disclosed herein include devices, systems, andmethods that provide retinal imaging-based gaze tracking. In someimplementations, a user's gaze is tracked based on a retinal imagingtechnique that selectively illuminates a subset of multiple lightsources to direct light towards one or more portions of the user'sretina. In some implementations, the subset of the light sources is usedin retinal imaging to provide better gaze tracking by reducing glare,reducing image saturation, reducing imaging computation, or reducingenergy consumption.

In some implementations, retinal imaging-based gaze tracking uses asubset of the light sources that are each configured to direct light toa different location (e.g., region) on the retina. Based on a map of theretina, a selected subset of light sources may be used during static eyegaze conditions (e.g., about 80% of the time). When the eye makes a bigmovement, all or most of the light sources of the multiple light sourcesmay be enabled to concurrently evaluate the whole retina (e.g., entireretinal field) to identify a new gaze direction or gaze landing point.When the new gaze direction is determined, the subset of light sourcesmay again be used. In some implementations, the subset of light sourcesof the multiple light sources is different for each of a plurality ofdifferent gaze directions. In some implementations, the multiple lightsources may use a single collimation lens or a collimating lens forevery individual light source. In some implementations, the multiplelight sources is an array of light sources (e.g., 2D/3D array).

In some implementations, unwanted glare (reflection) from other surfacesof the eye including the cornea and crystalline lens caused by specificones of the multiple light sources may be identified (e.g., duringenrollment) and reduced by disabling those specific light sources. Insome implementations, energy consumption may be reduced by disablingspecific light sources during retinal imaging-based gaze tracking.

In some implementations, a method includes selecting a subset of thelight sources, the subset of the light sources including less than allof the light sources. In some implementations, one or more of the subsetof the light sources is illuminated to direct light towards one or moreportions of the user's retina. Then, sensor data is received at asensor, where the sensor data corresponds to the light detected usingretinal imaging, and an eye characteristic is determinized based on thesensor data.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIG. 1 illustrates an example operating environment in accordance withsome implementations.

FIG. 2 illustrates an exemplary head mounted device (HMD) in accordancewith some implementations.

FIG. 3 is a block diagram that illustrates an exemplary tracking systemthat selectively uses a subset of multiple light sources to illuminateone or more portions of a retina for gaze tracking in accordance withsome implementations.

FIG. 4 is a diagram that illustrates glare from exemplary light sourcesused to illuminate a retina in accordance with some implementations.

FIG. 5 is a block diagram that illustrates an exemplary tracking systemthat selectively uses a subset of multiple light sources to illuminateone or more portions of a retina for gaze tracking in accordance withsome implementations.

FIG. 6 is a diagram that illustrates exemplary light sources near animage sensor used to illuminate a retina in accordance with someimplementations.

FIG. 7 is a flowchart illustrating an exemplary method of tracking auser's gaze based on a retinal imaging technique that selectively uses asubset of light sources to illuminate one or more portions of the retinain accordance with some implementations.

FIG. 8 illustrates an example electronic device in accordance with someimplementations.

In accordance with common practice, the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsor variants do not include all of the specific details described herein.Moreover, well-known systems, methods, components, devices and circuitshave not been described in exhaustive detail so as not to obscure morepertinent aspects of the example implementations described herein.

FIG. 1 illustrates an example operating environment 100 in whichelectronic device 120 is used in physical environment 105. A physicalenvironment refers to a physical world that people can interact withand/or sense without the aid of electronic systems. Physicalenvironments, such as a physical park, include physical articles, suchas physical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In the example of FIG. 1 , the device 120 is illustrated as a singledevice. Some implementations of the device 120 are hand-held. Forexample, the device 120 may be a mobile phone, a tablet, a laptop, andso forth. In some implementations, the device 120 is worn by a user. Forexample, the device 120 may be a watch, a head-mounted device (HMD), andso forth. In some implementations, functions of the device 120 areaccomplished via two or more devices, for example additionally includingan optional base station. Other examples include a laptop, desktop,server, or other such device that includes additional capabilities interms of power, CPU capabilities, GPU capabilities, storagecapabilities, memory capabilities, and the like. The multiple devicesthat may be used to accomplish the functions of the device 120 maycommunicate with one another via wired or wireless communications.

Various implementations disclosed herein include devices, systems, andmethods that implement retinal imaging-based gaze tracking. In someimplementations, illumination for retinal imaging-based gaze tracking isseparated into an array of illuminators, which each illuminate adifferent region on the retina. In some implementations, by turning on aspecific illuminator or a subset of the illuminators of the array ofilluminators, power consumption or glare is reduced during gazetracking. In some implementations, a specific illuminator or a subset ofthe illuminators of the array of illuminators is chosen based on a gazedirection of the user. The subset of the illuminators is less than allof the array of illuminators. For example, when the eye gaze is static,reduced illumination or more selective illumination may be used. In someimplementations, the array of illuminators includes multiple addressablelight sources such as vertical-cavity surface-emitting lasers (VCSEL) onSilicon, or light emitting diodes (LEDs).

FIG. 2 illustrates a block diagram of a HMD 200 in accordance with someimplementations. The head-mounted device 200 includes a housing 201 (orenclosure) that houses various components of the head-mounted device200. The housing 201 includes (or is coupled to) an eye pad 205 disposedat a proximal (to the user 10) end of the housing 201. In variousimplementations, the eye pad 205 is a plastic or rubber piece thatcomfortably and snugly keeps the head-mounted device 200 in the properposition on the face of the user 10 (e.g., surrounding the eye of theuser 10).

The housing 201 houses a display 210 that displays an image, emittinglight towards onto the eye of a user 10. In various implementations, thedisplay 210 emits the light through an eyepiece (not shown) thatrefracts the light emitted by the display 210, making the display appearto the user 10 to be at a virtual distance farther than the actualdistance from the eye to the display 210. For the user to be able tofocus on the display 210, in various implementations, the virtualdistance is at least greater than a minimum focal distance of the eye(e.g., 7 cm). Further, in order to provide a better user experience, invarious implementations, the virtual distance is greater than 1 meter.

Although FIG. 2 illustrates a head-mounted device 200 including adisplay 210 and an eye pad 205, in various implementations, thehead-mounted device 200 does not include a display 210 or includes anoptical see-through display without including an eye pad 205.

The housing 201 also houses a pupil assessment system including one ormore light sources 222, image sensor 224, and a controller 280. The oneor more light sources 222 emit light towards the eye of the user 10 thatreflects light (e.g., a directional beam) that can be detected by thesensor 224. Based on the reflections, the controller 280 can determinepupil characteristics of the user 10. As another example, the controller280 can determine a pupil center, a pupil size, gaze direction, or apoint of regard. Thus, in various implementations, the light is emittedby the one or more light sources 222, reflects off the eye of the user10, and is detected by the sensor 224. In various implementations, thelight from the eye of the user 10 is reflected off a hot mirror orpassed through an eyepiece before reaching the sensor 224.

The display 210 may emit light in a first wavelength range and the oneor more light sources 222 may emit light in a second wavelength range.Similarly, the sensor 224 may detects light in the second wavelengthrange. In various implementations, the first wavelength range is avisible wavelength range (e.g., a wavelength range within the visiblespectrum of approximately 400-700 nm) and the second wavelength range isa near-infrared wavelength range (e.g., a wavelength range within thenear-infrared spectrum of approximately 700-1400 nm).

In some implementations, one or more of the light sources 222 each pointto a different region on the retina. In some implementations, retinalimaging-based gaze tracking uses a subset of the light sources 222.Based on a map of the retina obtained during enrollment, the subset ofthe light sources 222 used for retinal imaging-based gaze tracking maybe used during static eye gaze conditions (e.g., about 80% of the time).When the eye makes a big movement, all or more than the subset of thelight sources 222 may be enabled to concurrently evaluate the wholeretina (e.g., entire retinal field) to identify a new gaze direction orgaze landing point. When the new gaze direction is determined, thesubset that is less than all of the light sources 222 may again be used.In some implementations, the subset of light sources 222 is differentfor each of a plurality of different gaze directions.

In some implementations, the one or more other light sources (not shown)emit light towards the eye of the user which reflects in the form of oneor more glints off the surface of the eye.

In various implementations, the sensor 224 is a frame/shutter-basedcamera that, at a particular point in time or multiple points in time ata frame rate, generates an image of the eye of the user 10. Each imageincludes a matrix of pixel values corresponding to pixels of the imagewhich correspond to locations of a matrix of light sensors of thecamera.

In various implementations, the camera 224 is an event camera comprisinga plurality of light sensors (e.g., a matrix of light sensors) at aplurality of respective locations that, in response to a particularlight sensor detecting a change in intensity of light, generates anevent message indicating a particular location of the particular lightsensor.

In various implementations, pupil characteristic assessment is used tofacilitate gaze tracking, which may be used to enable user interaction(e.g., the user 10 selects an option on the display 210 by looking atit), provide foveated rendering (e.g., present a higher resolution in anarea of the display 210 the user 10 is looking at and a lower resolutionelsewhere on the display 210), or reduce geometric distortion (e.g., in3D rendering of objects on the display 210).

FIG. 3 is a block diagram that illustrates an exemplary eye trackingsystem that selectively uses a subset of multiple light sources toilluminate one or more portions of a retina for gaze tracking inaccordance with some implementations. In some implementations, using asubset of the multiple light sources in retinal imaging provides bettergaze tracking by reducing glare, reducing image saturation, reducingimaging computation, or reducing energy consumption.

As shown in FIG. 3 , multiple lights sources 310, a collimator 320, abeam splitter 330 and an image sensor 340 form a retinal imaging-basedgaze tracking system. In some implementations, the multiple lightsources 310 include multiple addressable light sources 310 a, 310 b, . .. , 310 e that when enabled each illuminate different portions of aretina 352 of an eye 350.

As shown in FIG. 3 , the multiple lights sources 310 produce light thatis reflected off the beam splitter 330 at an angle to provide light thatilluminates different portions of the retina 352. The reflected lightfrom the retina 352 passes through the beam splitter 330 and is capturedby the image sensor 340 (e.g., IR camera). As shown in FIG. 3 , the onlyenabled light source 310 c of the light sources 310 uses the beamsplitter 330 to provide approximately on-axis illumination. The sameeffect of rendering a virtual location of a light source to besubstantially coaxial with the camera axis can be achieved usingdifferent types of waveguides, including using mirrors, fiber optics,diffractive optical elements, holographic elements, etc.

In some implementations, the subset of the multiple light sources 310are selectively turned on depending upon gaze direction. In someimplementations, selectively enabling a subset of the multiple lightsources 310 is related to the portion of the retina 350 that is viewableby the image sensor 340 in a current image, which is related to the gazedirection (e.g., eye orientation). In some implementations, the subsetof the multiple light sources 310 chosen for each different gazedirection is different.

Generally, the gaze direction of the eye 350 is stable or static 80% ofthe time. In some implementations, the static gaze direction of the eye350 allows for more selective illumination (e.g., less illumination orless energy) using the subset of the multiple light sources 310. In someimplementations, the subset of the multiple light sources 310 may beintermittently disabled or pulsed to further reduce energy consumption.Alternatively, an illumination strength of the subset of the multiplelight sources 310 may be reduced to further reduce energy consumption.In some implementations, the static gaze direction includes minormovements of the eye such as micro-saccades or the like. In someimplementations, the static gaze direction includes minor movements ofthe eye that are less than a movement threshold. In someimplementations, when the gaze direction of the eye 350 is static, asfew as a single light source of the multiple light sources 310 may beused to monitor or track the gaze direction of the eye 350. For example,two or more light sources 310 that actually direct light towards theretina 352 in a field of view (FOV) of the image sensor 340 for a gazedirection are selected as the subset of light sources 310. In someimplementations, instead of directing light towards the retina 352 atslightly different angles using all the multiple light sources 310,respectively, a better image of the retina 352 may result using thesubset of the multiple light sources 310.

In some implementations, selectively enabling a subset of the lightsources 310 based on gaze direction may conserve power, reduce glare, orreduce the number of ineffective illumination sources.

In some implementations, selective illumination (e.g., lessillumination) using the subset of the multiple light sources 310 reducesthe glare at the image sensor 340 caused by light (e.g., intended to beilluminating the retina 352) reflecting on a cornea 354 back to saturateportions of the image sensor 340. In some implementations, the lightreflecting off the cornea 354 may obscure or occlude the retina 352,which may interfere with gaze tracking.

In some implementations, glare 360 of reflected light off the cornea 354is reduced or prevented by selectively turning on/off different ones ofthe multiple light sources 310 when choosing the subset of light sourcesin order to obtain a retinal image that is not impacted by cornealreflections. In a similar fashion, glare of reflected light off acrystalline lens 356 of the eye 350 is reduced or prevented byselectively turning on/off different ones of the multiple light sources310 when choosing the subset of light sources in order to obtain aretinal image that is not impacted by reflections off the crystallinelens 356.

FIG. 4 is a diagram that illustrates glare from exemplary light sourcesused to illuminate a retina in accordance with some implementations. Asshown in FIG. 4 , glare 360 of reflected light off the cornea 354 maysaturate portions of an image of the eye 350 produced by the imagesensor 340. When the saturation of the image sensor 340 caused by theglare 360 aligns with the retina 352, the image of the retina 352 may beobstructed. As shown in FIG. 4 , significant glare 360 occurs whenilluminating portions of the retina 352 through an iris 359 of a pupil358 of the eye 350. In addition, a size of the pupil 358 varies amongpeople, and the glare 360 may cause increased occlusion of the retina352 with smaller pupils (e.g., the occlusion may be large enough tocover the entire pupil or iris).

In some implementations, one or more of the multiple light sources 310will illuminate portions of the retina 352 that have very few or nofeatures (e.g., blood vessels). During retinal imaging-based gazetracking, analyzing featureless portions of the retina 352 increasescomputations needed for retinal image processing without benefit (e.g.,without improving gaze tracking). In some implementations, when tryingto determine the orientation of the eye 350 from a current retinalimage, featureless portions of the retina 352 may be identified andignored, and corresponding ones of the multiple light sources 310 notselected for the subset of light sources (e.g., disabled). In someimplementations disabling ones of the subset of the multiple lightsources 310 that illuminate featureless portions of the retina 352 basedon gaze direction reduces energy consumption during retinalimaging-based gaze tracking.

Example operations of retinal imaging-based gaze tracking will now bedescribed. In some implementations, a first phase of retinalimaging-based gaze tracking is “enrollment”. In some implementations, asecond phase of retinal imaging-based gaze tracking is “gaze detection”or active gaze tracking.

In some implementations, enrollment is used to generate a map of theretina 352 based on gaze direction (e.g., eye orientation). Thus, whilewearing an electronic device such as an HMD, a user is instructed tolook in several different specific locations (e.g., left to right andtop to bottom) and an image of the retina while looking at each specificlocation is generated. In some implementations, while looking at each ofthe specific locations, any of the multiple light sources 310 that causeglare (e.g., the glare 360) are identified (e.g., for potentialexclusion). In some implementations, while looking at each of thespecific locations, any of the multiple light sources 310 thatilluminate featureless portions of the retina are identified (e.g., forpotential exclusion). Then, in some implementations, the individualimages (e.g., maps of portions of the retina) for each of the specificlocations are combined into sectional maps of the retina 352 or asingle, larger, combined map of the retina 352. During subsequentretinal imaging-based active gaze tracking, matching a current view of aportion of the retina 352 to the enrollment map of the retina (e.g.,individual maps, sectional maps, or the combined retina map), identifiesor determines a current gaze direction of the eye 350.

Initially, during retinal image based active gaze tracking, all or apredetermined sequence of the multiple light sources 310 are enabled andthe image sensor 340 takes a current image of the retina 352. Once thecurrent image of the retina 352 is identified in the combined retinamap, an eye orientation or a current gaze direction of the eye 350 isknown and a number of light sources 310 being used can be reduced (e.g.,the subset of light sources 310 identified for that gaze direction maybe used).

In some implementations, the reduced number of light sources in thesubset of light sources 310 may be used until there is a large movementof the eye 350 (e.g., over a threshold amount). Alternatively, in someimplementations, the subset of light sources 310 may be used until aresulting current image of the retina 352 does not match the map of theretina for that gaze direction. In some implementations, when thecurrent gaze direction changes, the image sensor 340 takes images of theretina 352 while all of the multiple light sources 310 are enabled untilthe current retinal image is again matched to the enrollment retina map,and the subset of light sources 310 for that gaze direction are used. Insome implementations, the subset of light sources excludes/disablesunused lights, lights that cause glare, lights that illuminatefeatureless regions, or lights that are redundant (e.g., within themultiple light sources 310) to reduce the number of lights in the subsetof light sources 310 used for a specific gaze.

FIG. 5 is a block diagram that illustrates an exemplary eye trackingsystem that selectively uses a subset of multiple light sources toilluminate one or more portions of a retina for gaze tracking inaccordance with some implementations. As shown in FIG. 5 , the onlyenabled light source 310 c of the light sources 310 uses the beamsplitter 330 to provide off-axis illumination. Some implementationscombine on-axis and off-axis illumination-based eye assessments.

In some implementations, the multiple light sources 310 may use a singlecollimation lens 320 or a collimating lens for each of the individuallight sources 310 a, 310 b, . . . , 310 e. In some implementations, themultiple light sources 310 are directional light sources. In someimplementations, the multiple light sources 310 are a 1D array of lightsources. In some implementations, the multiple light sources 310 are a2D array of light sources (e.g., a ring or rectangle around the imagesensor). In some implementations, the multiple light sources 310 are a3D array of light sources (e.g., not arranged in a single plane). Insome implementations, the light sources 310 include several hundred orseveral thousand VCSELs.

FIG. 6 is a diagram that illustrates exemplary light sources near animage sensor used to illuminate a retina in accordance with someimplementations. As shown in FIG. 6 , the multiple lights sources 610produce light that illuminates different portions of the retina 352 tobe captured by the image sensor 340 (e.g., IR camera). In someimplementations, a subset of multiple light sources 610 illuminate oneor more portions of the retina 352 for gaze tracking. In someimplementations, the multiple lights sources 610 are directional lightsources (e.g., LEDs). As shown in FIG. 6 , the only enabled light source610 g of the light sources 310 is near the optics of the image sensor340 and provides approximately on-axis retinal illumination according tosome implementations.

FIG. 7 is a flowchart illustrating an exemplary method of tracking auser's gaze based on a retinal imaging technique that selectively uses asubset of light sources to illuminate one or more portions of the retinain accordance with some implementations. In some implementations, thesubset of the light sources is less than all the light sources andprovides better gaze tracking using retinal imaging by reducing glare,reducing image saturation, reducing imaging computation, or reducingenergy consumption. In some implementations, the method 700 is performedby a device (e.g., electronic device 120, 200 of FIGS. 1-2 ). The method700 can be performed using an electronic device or by multiple devicesin communication with one another. In some implementations, the method700 is performed by processing logic, including hardware, firmware,software, or a combination thereof. In some implementations, the method700 is performed by a processor executing code stored in anon-transitory computer-readable medium (e.g., a memory). In someimplementations, the method 700 is performed by an electronic devicehaving a processor and light sources.

At block 710, the method 700 selects a subset of the light sources, thesubset of the light sources including less than all of the lightsources. In some implementations, the light sources are a 1D, 2D or 3Darray of light sources. In some implementations, the light sources areone or more rings or polygons of light sources. In some implementations,the light sources are LEDs or VCSELs. In some implementations, the lightsources are individually addressable (e.g., enabled or disabled).

At block 720, the method 700 illuminates one or more portions of aretina by producing light using the subset of the light sources (e.g.,retinal imaging). In some implementations, the light sources illuminatedifferent locations on the retina. In some implementations, the lightsources may provide lights at different angles to illuminate differentlocations on the retina. In some implementations, the light sources mayprovide directional lights to illuminate different locations on theretina. In some implementations, the light sources use a singlecollimation lens. In some implementations, the light sources use aseparate collimating lens for every individual light source. In someimplementations, the subset of the light sources illuminate a smallportion of the retina.

At block 730, the method 700 receives sensor data at a sensor, thesensor data corresponding to the light detected or obtained duringretinal imaging. In some implementations, an image sensor receives imagesensor data that corresponds to the light reflected or scattered fromthe retina. In some implementations, an IR camera receives IR image datacorresponding to the light reflected from the retina. In someimplementations, the sensor receives an IR image of the retinacorresponding to the light reflected from the retina. In someimplementations, the sensor is part of a scanning apparatus or thesensor data is obtained using a scanning apparatus. For example, thereceived image may be formed using a point-by-point scanning apparatusor a line-by-line scanning apparatus. In some implementations, retinalimaging is performed using a scanning mirror and a single photodetector.For example, the scanning mirror and the single photodetector may be ina confocal arrangement.

At block 740, the method 700 determines an eye characteristic based onthe sensor data. In some implementations, the eye characteristicincludes an eye gaze direction, an eye orientation, etc. based on thesensor data. In some implementations, the eye characteristic isdetermined by comparing an image of the retina (the sensor data) to apreviously-generated map of the retina. In some implementations, thepreviously-generated map of the retina may be generated during anenrollment process.

In some implementations, the subset of the light sources is selectedbased on a previously-detected eye characteristic (e.g., based on aninitial/prior gaze direction or eye orientation). In someimplementations, the previously-detected eye characteristic isdetermined using light produced using more than the subset of lightsources (e.g., at least one more light source than the subset of lightsources) and a map of the retina. For example, all of the light sourcesmay be turned on to illuminate the entire retinal field when a HMD userturns on gaze tracking at the HMD.

In some implementations, the method at block 740 continues to receivethe sensor data and determine the eye characteristic until an eye changeevent is detected. In some implementations, the eye change event is achange in eye orientation or gaze direction that exceeds a threshold. Insome implementations, upon detecting the eye change event, the eyecharacteristic is re-determined using light produced by more than thesubset of light sources (e.g., most of the light sources, all of thelight sources, or at least one more light source than the subset oflight sources) and a map of the retina.

In some implementations, one or more additional criterion may be used toselect the subset of light sources. In some implementations, the subsetof light sources is selected by determining that a first light source ofthe light sources is producing an imaging defect (e.g., glare,saturation, etc.) in retinal images obtained via the sensor, andexcluding the first light source from the subset of light sources. Insome implementations, the subset of light sources is selected byselecting a second light source of the light sources based on theretinal pattern (e.g., sparseness of features like blood vessels,minimal number of features, or featureless) illuminated by the secondlight source, and excluding the second light source from the subset oflight sources. In some implementations, the one or more additionalcriterion are based on the eye characteristic like gaze direction. Insome implementations, the one or more additional criterion aredetermined during enrollment.

In some implementations, finding the gaze direction of the eye initiallyor the eye change event uses more/most/all of the light sources. In someimplementations, more/most/all of the light sources are used toilluminate a larger portion of the retina. Then, once the gaze directionis determined, a reduced set of the light sources is selected (e.g., thesubset of light sources) and used to track or monitor small eyemovements until the eye change event occurs again.

In some implementations, the method 700 further includes retinalimaging-based gaze tracking using both eyes of user. In someimplementations, the sensor data may be a still image, series of images,video, etc., which may include depth information such as a correspondingdepth map.

FIG. 8 is a block diagram of an example device 800. Device 800illustrates an exemplary device configuration for the device 120. Whilecertain specific features are illustrated, those skilled in the art willappreciate from the present disclosure that various other features havenot been illustrated for the sake of brevity, and so as not to obscuremore pertinent aspects of the implementations disclosed herein. To thatend, as a non-limiting example, in some implementations the electronicdevice 800 includes one or more processing units 802 (e.g.,microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, or thelike), one or more input/output (I/O) devices and sensors 806, one ormore communication interfaces 808 (e.g., USB, FIREWIRE, THUNDERBOLT,IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR,BLUETOOTH, ZIGBEE, SPI, I2C, or the like type interface), one or moreprogramming (e.g., I/O) interfaces 810, one or more displays 812, one ormore interior or exterior facing sensor systems 814, a memory 820, andone or more communication buses 804 for interconnecting these andvarious other components.

In some implementations, the one or more communication buses 804 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices andsensors 806 include at least one of an inertial measurement unit (IMU),an accelerometer, a magnetometer, a gyroscope, a thermometer, one ormore physiological sensors (e.g., blood pressure monitor, heart ratemonitor, blood oxygen sensor, blood glucose sensor, etc.), one or moremicrophones, one or more speakers, a haptics engine, one or more depthsensors (e.g., a structured light, a time-of-flight, or the like), orthe like.

In some implementations, the one or more displays 812 are configured topresent content to the user. In some implementations, the one or moredisplays 812 correspond to holographic, digital light processing (DLP),liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organiclight-emitting field-effect transitory (OLET), organic light-emittingdiode (OLED), surface-conduction electron-emitter display (SED),field-emission display (FED), quantum-dot light-emitting diode (QD-LED),micro-electromechanical system (MEMS), or the like display types. Insome implementations, the one or more displays 812 correspond todiffractive, reflective, polarized, holographic, etc. waveguidedisplays. For example, the electronic device 800 may include a singledisplay. In another example, the electronic device 800 includes adisplay for each eye of the user.

In some implementations, the one or more interior or exterior facingsensor systems 814 include an image capture device or array thatcaptures image data or an audio capture device or array (e.g.,microphone) that captures audio data. The one or more image sensorsystems 814 may include one or more RGB cameras (e.g., with acomplimentary metal-oxide-semiconductor (CMOS) image sensor or acharge-coupled device (CCD) image sensor), monochrome cameras, IRcameras, event-based cameras, or the like. In various implementations,the one or more image sensor systems 814 further include an illuminationsource that emits light such as a flash. In some implementations, theone or more image sensor systems 814 further include an on-camera imagesignal processor (ISP) configured to execute a plurality of processingoperations on the image data.

The memory 820 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome implementations, the memory 820 includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid-state storagedevices. The memory 820 optionally includes one or more storage devicesremotely located from the one or more processing units 802. The memory820 comprises a non-transitory computer readable storage medium.

In some implementations, the memory 820 or the non-transitory computerreadable storage medium of the memory 820 stores an optional operatingsystem 830 and one or more instruction set(s) 840. The operating system830 includes procedures for handling various basic system services andfor performing hardware dependent tasks. In some implementations, theinstruction set(s) 840 include executable software defined by binaryinformation stored in the form of electrical charge. In someimplementations, the instruction set(s) 840 are software that isexecutable by the one or more processing units 802 to carry out one ormore of the techniques described herein.

In some implementations, the instruction set(s) 840 include a retinalimage generator 842 that is executable by the processing unit(s) 802 tocapture sensor data representing a retina of a user of the device 800according to one or more of the techniques disclosed herein.

In some implementations, the instruction set(s) 840 include an eyecharacteristic detector 844 that is executable by the processing unit(s)802 to determine a gaze direction of the like of the user of theelectronic device according to one or more of the techniques disclosedherein. In some implementations, the eye characteristic detector 844 isexecuted to compare a current retinal image with a map of the retina ofthe user of the electronic device.

In some implementations, the instruction set(s) 840 include a lightsource controller 846 that is executable by the processing unit(s) 802to determine a subset of retinal imaging light sources to enable basedon the gaze direction of the user of the electronic device according toone or more of the techniques disclosed herein. In some implementations,the subset of retinal imaging light sources is one or more and less thanall of retinal imaging light sources.

Although the instruction set(s) 840 are shown as residing on a singledevice, it should be understood that in other implementations, anycombination of the elements may be located in separate computingdevices. FIG. 8 is intended more as a functional description of thevarious features which are present in a particular implementation asopposed to a structural schematic of the implementations describedherein. As recognized by those of ordinary skill in the art, items shownseparately could be combined and some items could be separated. Forexample, actual number of instruction sets and the division ofparticular functions and how features are allocated among them will varyfrom one implementation to another and, in some implementations, dependsin part on the particular combination of hardware, software, or firmwarechosen for a particular implementation.

In some implementations, the electronic device 800 is a head mountedsystem including one or more speaker(s) and an integrated opaquedisplay. Alternatively, the head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). Rather than anopaque display, the head mounted system may have a transparent ortranslucent display. The transparent or translucent display may have amedium through which light representative of images is directed to aperson's eyes. The display may utilize digital light projection, OLEDs,LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, orany combination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

It will be appreciated that the implementations described above arecited by way of example, and that the present invention is not limitedto what has been particularly shown and described hereinabove. Rather,the scope includes both combinations and sub combinations of the variousfeatures described hereinabove, as well as variations and modificationsthereof which would occur to persons skilled in the art upon reading theforegoing description and which are not disclosed in the prior art.

As described above, one aspect of the present technology is thegathering and use of physiological data to improve a user's experienceof an electronic device. The present disclosure contemplates that insome instances, this gathered data may include personal information datathat uniquely identifies a specific person or can be used to identifyinterests, traits, or tendencies of a specific person. Such personalinformation data can include physiological data, demographic data,location-based data, telephone numbers, email addresses, home addresses,device characteristics of personal devices, or any other personalinformation.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used toimprove the content viewing experience. Accordingly, use of suchpersonal information data may enable calculated control of theelectronic device. Further, other uses for personal information datathat benefit the user are also contemplated by the present disclosure.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information and/or physiological data willcomply with well-established privacy policies and/or privacy practices.In particular, such entities should implement and consistently useprivacy policies and practices that are generally recognized as meetingor exceeding industry or governmental requirements for maintainingpersonal information data private and secure. For example, personalinformation from users should be collected for legitimate and reasonableuses of the entity and not shared or sold outside of those legitimateuses. Further, such collection should occur only after receiving theinformed consent of the users. Additionally, such entities would takeany needed steps for safeguarding and securing access to such personalinformation data and ensuring that others with access to the personalinformation data adhere to their privacy policies and procedures.Further, such entities can subject themselves to evaluation by thirdparties to certify their adherence to widely accepted privacy policiesand practices.

Despite the foregoing, the present disclosure also contemplatesimplementations in which users selectively block the use of, or accessto, personal information data. That is, the present disclosurecontemplates that hardware or software elements can be provided toprevent or block access to such personal information data. For example,in the case of user-tailored content delivery services, the presenttechnology can be configured to allow users to select to “opt in” or“opt out” of participation in the collection of personal informationdata during registration for services. In another example, users canselect not to provide personal information data for targeted contentdelivery services. In yet another example, users can select to notprovide personal information, but permit the transfer of anonymousinformation for the purpose of improving the functioning of the device.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, content can beselected and delivered to users by inferring preferences or settingsbased on non-personal information data or a bare minimum amount ofpersonal information, such as the content being requested by the deviceassociated with a user, other non-personal information available to thecontent delivery services, or publicly available information.

In some embodiments, data is stored using a public/private key systemthat only allows the owner of the data to decrypt the stored data. Insome other implementations, the data may be stored anonymously (e.g.,without identifying and/or personal information about the user, such asa legal name, username, time and location data, or the like). In thisway, other users, hackers, or third parties cannot determine theidentity of the user associated with the stored data. In someimplementations, a user may access their stored data from a user devicethat is different than the one used to upload the stored data. In theseinstances, the user may be required to provide login credentials toaccess their stored data.

Numerous specific details are set forth herein to provide a thoroughunderstanding of the claimed subject matter. However, those skilled inthe art will understand that the claimed subject matter may be practicedwithout these specific details. In other instances, methods,apparatuses, or systems that would be known by one of ordinary skillhave not been described in detail so as not to obscure claimed subjectmatter.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing the terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provides a resultconditioned on one or more inputs. Suitable computing devices includemultipurpose microprocessor-based computer systems accessing storedsoftware that programs or configures the computing system from a generalpurpose computing apparatus to a specialized computing apparatusimplementing one or more implementations of the present subject matter.Any suitable programming, scripting, or other type of language orcombinations of languages may be used to implement the teachingscontained herein in software to be used in programming or configuring acomputing device.

Implementations of the methods disclosed herein may be performed in theoperation of such computing devices. The order of the blocks presentedin the examples above can be varied for example, blocks can bere-ordered, combined, or broken into sub-blocks. Certain blocks orprocesses can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor value beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

It will also be understood that, although the terms “first,” “second,”etc. may be used herein to describe various objects, these objectsshould not be limited by these terms. These terms are only used todistinguish one object from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularimplementations only and is not intended to be limiting of the claims.As used in the description of the implementations and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “or” as used hereinrefers to and encompasses any and all possible combinations of one ormore of the associated listed items. It will be further understood thatthe terms “comprises” or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,objects, or components, but do not preclude the presence or addition ofone or more other features, integers, steps, operations, objects,components, or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

The foregoing description and summary of the invention are to beunderstood as being in every respect illustrative and exemplary, but notrestrictive, and the scope of the invention disclosed herein is not tobe determined only from the detailed description of illustrativeimplementations, but according to the full breadth permitted by patentlaws. It is to be understood that the implementations shown anddescribed herein are only illustrative of the principles of the presentinvention and that various modification may be implemented by thoseskilled in the art without departing from the scope and spirit of theinvention.

1. A method comprising: at an electronic device having a processor and aplurality of light sources configured to direct light at a retina:selecting a subset of the light sources, the subset of the light sourcescomprising less than all of the light sources; illuminating the subsetof the light sources to direct light toward one or more portions of theretina; receiving sensor data at a sensor, the sensor data correspondingto the light reflected from the retina; and determining an eyecharacteristic based on the sensor data.
 2. The method of claim 1,wherein the eye characteristic is a gaze direction or eye orientation.3. The method of claim 1, wherein the subset of the light sources isselected based on a previously-detected eye characteristic.
 4. Themethod of claim 3, wherein the previously-detected eye characteristic isdetermined using light produced using more than the subset of lightsources and a map of the retina.
 5. The method of claim 1 furthercomprising continuing to receive the sensor data and determine the eyecharacteristic until an eye change event is detected, the eye changeevent corresponding to a change in eye orientation or gaze directionthat exceeds a threshold.
 6. The method of claim 5 further comprising,upon detecting the eye change event, redetermining the eyecharacteristic using light produced using more than the subset of lightsources and a map of the retina.
 7. The method of claim 1, wherein thesubset of lights sources is selected by: determining that a first lightsource of the light sources is producing an imaging defect in retinalimages obtained via the sensor; and in accordance with determining thatthe light source is producing the imaging defect, excluding the firstlight source from the subset of light sources.
 8. The method of claim 7,wherein the imaging defect comprises glare or saturation.
 9. The methodof claim 1, wherein the subset of lights sources is selected by:selecting a first light source of the light sources based on the retinalpattern illuminated by the first light source; and excluding the firstlight source from the subset of light sources.
 10. The method of claim1, wherein each light source of the light sources directs light toward adifferent region on the retina.
 11. The method of claim 1, wherein thelight is infrared (IR) light.
 12. A device comprising: light sourcesconfigured to direct light at a retina; a sensor configured providesensor data; a processor; and a computer-readable storage mediumcomprising instructions that upon execution by the processor cause thesystem to perform operations, the operations comprising: selecting asubset of the light sources, the subset of the light sources comprisingless than all of the light sources; illuminating the subset of the lightsources to direct light toward one or more portions of the retina;receiving the sensor data at the sensor, the sensor data correspondingto the light reflected from the retina; and determining an eyecharacteristic based on the sensor data.
 13. The device of claim 12,further comprising a single collimator through which light from all ofthe light sources is directed.
 14. The device of claim 12, wherein eachlight source of the light sources uses a separate collimator.
 15. Thedevice of claim 12, wherein each light source of the light sources is avertical-cavity surface-emitting laser (VCSEL).
 16. The device of claim12, wherein at least one of the light sources is aligned approximatelyon axis with an optical axis of the sensor.
 17. The device of claim 16,wherein approximately on-axis illumination is produced via a lightsource sufficiently near optics of the sensor such that the light sourceproduces a light ray that provides a reflection off the retina that iscaptured by the optics of sensor.
 18. The device of claim 17, whereinthe approximately on-axis illumination is produced via a light sourcering around optics of the sensor.
 19. The device of claim 12, whereineach light source of the light sources illuminates a different region onthe retina.
 20. The device of claim 12, wherein the electronic device isa head-mounted device (HMD).
 21. A non-transitory computer-readablestorage medium, storing program instructions computer-executable on acomputer to perform operations comprising: selecting a subset of lightsources configured to direct light at a retina, the subset of lightsources comprising less than all of the light sources; illuminating thesubset of the light sources to direct light toward one or more portionsof the retina; receiving sensor data at a sensor, the sensor datacorresponding to the light reflected from the retina; and determining aneye characteristic based on the sensor data.